Skip to content

HBASE-24356 Move hbase-connectors to hbase-thirdparty-4.0.1#89

Closed
taklwu wants to merge 2 commits intoapache:masterfrom
taklwu:HBASE-24356
Closed

HBASE-24356 Move hbase-connectors to hbase-thirdparty-4.0.1#89
taklwu wants to merge 2 commits intoapache:masterfrom
taklwu:HBASE-24356

Conversation

@taklwu
Copy link
Contributor

@taklwu taklwu commented Jan 11, 2022

  • upgrade spark version to 2.4.8
  • upgrade hbase version to 2.4.9
  • upgrade hadoop2 version to 2.10.0
  • upgrade hadoop3 version to 3.1.2
  • upgrade checkstyle to 8.28
  • update related client APIs after switching to 2.4.9

HBASE-24356 Move hbase-connectors to hbase-thirdparty-4.0.1
358bd65 
- upgrade spark version to 2.4.8
- upgrade hbase version to 2.4.9
- upgrade hadoop2 version to 2.10.0
- upgrade hadoop3 version to 3.1.2
- upgrade checkstyle to 8.28
- update related client APIs after switching to 2.4.9
@taklwu
Copy link
Contributor Author

taklwu commented Jan 11, 2022

I'm still in the progress of manually testing this hbase-connector with the following active hbase releases by running IntegrationTestSparkBulkLoad. and after this change I don't think the hbase-connector could support hbase-2.2.x

  1. hbase-2.4.9
  2. hbase-2.3.7
  3. optional hbase-2.2.7 (I'm not sure if this will work)

@Apache-HBase
Copy link

Apache-HBase commented Jan 11, 2022

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 9s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 spotbugs 0m 0s spotbugs executables are not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ master Compile Tests _
+0 🆗 mvndep 0m 30s Maven dependency ordering for branch
+1 💚 mvninstall 1m 22s master passed
+1 💚 compile 0m 47s master passed
+1 💚 javadoc 0m 51s master passed
+1 💚 scaladoc 0m 0s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 14s Maven dependency ordering for patch
+1 💚 mvninstall 0m 47s the patch passed
+1 💚 compile 0m 49s the patch passed
+1 💚 javac 0m 49s the patch passed
+1 💚 scalac 0m 49s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 1s The patch has no ill-formed XML file.
+1 💚 javadoc 0m 47s the patch passed
+1 💚 scaladoc 0m 0s the patch passed
_ Other Tests _
+1 💚 unit 7m 29s root in the patch passed.
15m 41s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-89/1/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR #89
Optional Tests dupname javac javadoc unit spotbugs findbugs compile xml scalac scaladoc
uname Linux 3f2e8c980492 5.4.0-1054-aws #57~18.04.1-Ubuntu SMP Thu Jul 15 03:21:36 UTC 2021 x86_64 GNU/Linux
Build tool hb_maven
Personality dev-support/jenkins/hbase-personality.sh
git revision master / 9a839b2
Default Java Oracle Corporation-1.8.0_282-b08
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-89/1/testReport/
Max. process+thread count 932 (vs. ulimit of 12500)
modules C: kafka/hbase-kafka-proxy spark spark/hbase-spark . U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-89/1/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

Apache-HBase commented Jan 11, 2022

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 0m 44s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 spotbugs 0m 0s spotbugs executables are not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ master Compile Tests _
+0 🆗 mvndep 0m 43s Maven dependency ordering for branch
+1 💚 mvninstall 3m 5s master passed
+1 💚 compile 0m 52s master passed
+1 💚 javadoc 0m 50s master passed
+1 💚 scaladoc 0m 0s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 16s Maven dependency ordering for patch
+1 💚 mvninstall 0m 48s the patch passed
+1 💚 compile 0m 50s the patch passed
+1 💚 javac 0m 50s the patch passed
+1 💚 scalac 0m 50s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 xml 0m 2s The patch has no ill-formed XML file.
+1 💚 javadoc 0m 49s the patch passed
+1 💚 scaladoc 0m 0s the patch passed
_ Other Tests _
+1 💚 unit 7m 48s root in the patch passed.
17m 52s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-89/2/artifact/yetus-precommit-check/output/Dockerfile
GITHUB PR #89
Optional Tests dupname javac javadoc unit spotbugs findbugs compile xml scalac scaladoc
uname Linux 84c2d71e2054 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 GNU/Linux
Build tool hb_maven
Personality dev-support/jenkins/hbase-personality.sh
git revision master / 9a839b2
Default Java Oracle Corporation-1.8.0_282-b08
Test Results https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-89/2/testReport/
Max. process+thread count 912 (vs. ulimit of 12500)
modules C: kafka/hbase-kafka-proxy spark spark/hbase-spark . U: .
Console output https://ci-hadoop.apache.org/job/HBase/job/HBase-Connectors-PreCommit/job/PR-89/2/console
versions git=2.20.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@taklwu
Copy link
Contributor Author

taklwu commented Jan 12, 2022 •
edited
Loading

I tested the following and they all passed with this change

### passed 
mvn clean install -Dhbase.version=2.3.7 -Dhadoop.profile=3.0 -Dhadoop-three.version=3.1.2
mvn clean install -Dhbase.version=2.4.9 -Dhadoop.profile=3.0 -Dhadoop-three.version=3.1.2 -DskipTests
mvn clean install -Dhadoop.profile=3.0 -Dhadoop-three.version=3.1.2 -Dspark.version=2.4.8 -Dscala.version=2.12.15 -Dscala.binary.version=2.12 -Dsurefire.rerunFailingTestsCount=3 -Prelease
mvn clean install -Dhadoop.profile=3.0 -Dhadoop-three.version=3.1.2 -Dspark.version=3.1.2 -Dscala.version=2.12.15 -Dscala.binary.version=2.12 -Dsurefire.rerunFailingTestsCount=3 -Prelease

# below should help to say the this change run on an older thirdparty
mvn clean install -Dhbase-thirdparty.version=3.3.0 -Dhbase.version=2.4.9 -Dhadoop.profile=3.0 -Dhadoop-three.version=3.1.2

@ianawilson ianawilson mentioned this pull request Jan 12, 2022
Merged
@taklwu
Copy link
Contributor Author

taklwu commented Jan 12, 2022

@ianawilson found another thread is doing similar work but want to set hadoop3, spark 3 and scala 2.12 as default, let's see if they can merge my changes into it and we can call this change as duplicate.

@taklwu
Copy link
Contributor Author

taklwu commented Jan 13, 2022

#88 has all the changes that we need from this PR, closing it.

@taklwu taklwu closed this Jan 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@meszibalu meszibalu Awaiting requested review from meszibalu

@petersomogyi petersomogyi Awaiting requested review from petersomogyi

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@taklwu @Apache-HBase